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ABSTRACT 



This paper focuses on the development of instruments to 



measure technology integration in learning environments. The model used as 
the basis for these instruments is the Technology Effectiveness Framework 
developed at the North Central Regional Educational Laboratory and described 
in "Designing Learning and Technology for Educational Reform" (B. Jones, J. 
Nowakowski, C. Rasmussen, and G. Valdez, 1994) . Based on this model, the 
following instruments were developed: (1) a classroom observation instrument; 

(2) a teacher report of classroom observation; (3) teacher beliefs and 
practices instrument; (4) teacher technology use instrument; (5) student 
technology use instrument; and (6) a survey of technology infrastructures. 
Participants were teachers at 5 high schools at which 35 classroom 
observations were performed. After pilot testing of the instruments, the 
instruments were refined. The six instruments work together to produce a 
picture of how a teacher combines pedagogy and technology in a classroom with 
access to technology. The instruments are attached. (Author/SLD) 
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Abstract 

■ This paper focuses on the development of instruments to measure technology integration in learning 
environments. The model used as the basis for these instruments is the Technology Effectiveness Framework 
developed at the North Central Regional Education Laboratory and described in Designing Learning and 
Technologyjbr Educational Reform (Jones, Nowakowski, Rasmussen, & Valdez, 1994). Based on this model the 
following instruments were developed: 1) Classroom Observation instrument, 2) Teacher Report of Classroom 

Observation, 3) Teacher Beliefs and Practices, 4) Teacher Technology Use instrument, 5) Student Technology Use 
instrument, and 6) Survey of Technology Infrastructures. 
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As computers are added to classrooms across the nation, the inevitable cry arises for evidence that this 
investment pays off in increased student learning. In an attempt to produce such evidence, Kulik and Kulik (1991) 
performed a meta-analysis of 254 studies that looked at the impact of computer-based instruction (CB1). They 
concluded that ... “CBI generally produces positive effects on students. These effects include modest increases in 
student test scores and positive attitudes towards technology and teaching” (p. 80). Researchers also reported that 
use of technology decreased the amount of time needed for instruction. 

Later, Hadley and Sheingold (1993) studied the ways in which teachers integrated computers into their 
classrooms. “Many of the teachers (88%) reported that such integration caused changes in their teaching practices 
including: increased expectations for students, increased individualization for students, conversion from a teacher- 
centered to a student-centered classroom, an increase in collaborative learning, and less time spent lecturing 1 ’ 
(p.275). They also offered the observation that the impact of technology is dependent upon how it is used in the 
classroom. 

Recent efforts to prescribe standards can be viewed as a way of establishing what ideal practice looks like. 

The benefit of establishing ideal practice is that it enables researchers to measure how far actual practice may be 
from the ideal. This provides a means of describing the variability of the learning environment from classroom to 
classroom. 

Hirumi and Grau (1996) conducted a review of state standards, textbooks and journal articles that described 
necessary technology skills for teachers. They found little agreement with only 1 skill (basic computer operations) 
being listed by more than 70% of the sources. 

The International Society for Technology in Education (ISTE) published National Education Technology 
Standards for Students (1998). Within that document they assert, “Certain conditions are necessary for schools to 
effectively use technology for learning, teaching and educational management” (p. 26). These necessary conditions 
include the support of “new learning environments” that would incorporate student centered learning, collaboration, 
use of multimedia, inquiry based learning, critical thinking, real-world activities, multiple paths for progression, and 
information exchange. 

Jones, Valdez, Nowakowski and Rasmussen (1994) of the North Central Regional Educational Laboratory 
(NCREL) have chosen a two-pronged approach to describe the learning environment of the school. They have developed 
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an “Effective Technology” framework which describes the classroom environment in terms of engaged learning 
characteristics and the use of high performance technologies. These indicators include many of the same elements as the 
ISTE “new learning environments”. Jones, Valdez, Nowakowski and Rasmussen, assert that “high-performance 
technology adds very substantial, qualitative differences to the learning environment that cannot be attained without that 
technology”(p. 4). 

Method 

This study grew out of a larger, multi-year investigation of technology integration in Iowa schools. During 
the first year of the study, thirty Iowa high schools were selected for a qualitative research effort to describe how 
technology was being used in high schools. This study resulted in thirty case studies which were used to select the 
five top technology-using schools from the thirty original participants. The selections were based on the Levels of 
Use and Stages of Concern scores using the Concerns Based Adoption Model by Gene Hall (1974) for each school 
as well as an examination of the case studies. This is reported in Selected perceptions of teachers and computer 
technology integration in Iowa high schools fMantemach-Wigans. L. K M 1999) 

Participants in the current study were teachers at the five high schools identified as exemplary in 
technology integration during the 1997-1998 Star Schools project year, and teachers at five elementary schools from 
the same districts. For each school visited, the goal was to perform announced observations in two classrooms and 
drop-in observations in two classrooms. The final total for classroom observations was 35 of which 15 were 
unannounced observations. Copies of the teacher’s beliefs and practices instrument was sent to teacher’s who had 
agreed to have their classes observed. Researchers gave the forms to the teachers who were visited during drop-in 
observations. 

Teachers and staff other than those who participated in the classroom observations were involved in the 
data collection process. The Survey of Technology Infrastructures was completed by the district technology 
coordinators. The Teacher Technology Use instrument was completed by teachers nominated by their principals as 
the top technology using teachers in their schools. 

Results 

1 • The Classroom Observation (CO) instrument consists of 21 indicators. Nineteen of these indicators are rated 0 - 
3. the last two are Technology Used and Software Used which are simply lists of the items observed in use during the 
lesson. 
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Internal Reliability Measure: The CO instrument was calculated to have a Cronbach Alpha of .88. 

2. The Teacher Beliefs^and Practices (TRI) consists of eleven open-ended questions designed to elicit teachers' 
beliefs and practices. Each question is scored with a separate rubric. Each rubric has a range of 0-3 points. 

Internal Reliability Measure: The TRI instrument was calculated to have a Cronbach Alpha of .47. This is 
a marginally acceptable level considering that the instrument consists of only 1 1 items which are all open response 
questions and the number of cases is low (23). 

Inter-rater Reliability: The 22 Teacher Beliefs and Practices were scored by 3 different researchers. The 
scores of the researchers were then compared to assess inter-rater reliability. Using a standard that a difference in 
scores that equals 1 or less is a match and a difference greater than one is a miss, the scores were compared for 
percent agreement. The percent agreement ranged from a low of 86% to a high of 90% with an average of 88% 
agreement. 

3. The Survey of Technology Infrastructure (STI) consists of 13 open-ended questions that are scored with separate 
rubrics. 

Internal Reliability Measure: The STI instrument was calculated to have a Cronbach Alpha of .83. 

Inter-rater Reliability: The 10 STI instruments were scored by 3 different researchers. The scores of the 
researchers were then compared to assess inter-rater reliability. Using a standard that a difference in scores that 
equals 1 or less is a match and a difference greater than one is a miss, the scores were compared for percent 
agreement. The percent agreement ranged from a low of 89% to a high of 95% with an average of 92% agreement. 

4. The Teacher Technology Use Survey (TTUS) consists of 91 questions organized into 4 different sub-scales. The 
sub-scales examine: how many times a teacher performs a specific technology-related task in a week, how many 
hours a teacher spends in these activities weekly, what proportion of the units taught annually include specific uses 
of technology, and what proportion of specific student activities are performed using technology. An open response 
question asked how technology has allowed the teacher to change teaching methodologies. Teachers are also asked 
to indicate if they need more training in specific areas and whether or not access issues prevent them from utilizing 
specific activities in the classroom. 
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Reliability Measure: This instrument has a calculated Cronbach Alpha of .86. 

Discussion 



Instrument Revisions 



Following the pilot testing of these instruments, a number of changes were made. Both the Survey of 
Technology Infrastructures instrument and the Teacher Beliefs and Practices were rewritten as forced-choice survey 
instruments. The responses to the open-ended questions which were obtained during the pilot provided the 
alternatives for the forced-choice items. Other changes were minor and consisted of removing duplicates, adding 
missing elements, or changing the layout of the instruments. 

Added Instrumentation 

In order to establish the validity of the classroom observations, a Teacher Classroom Observation 
instrument was added to the battery. This instrument enables both the observer and the teacher to report on what 
happened during the observation. The agreement between the two scores will provide the basis for a cross 
validation between what the teacher is attempting to do in the classroom in terms of providing engaged learning 
opportunities and what is observed by a second party visiting the classroom. 

An additional subsection has been added to the Student Technology Use instrument. This subsection is 
designed to assess where students encounter computers and graphing calculators (home or school), how many times 
and how many hours a week they spend in such pursuits, and how confident they are in their ability to use 
technology in different ways (a self-efficacy measure). 

Instrument Function 

These six instruments work together to provide a picture of how a teacher combines pedagogy and 
technology in a classroom where access to technology is available. The Teacher Beliefs and Practices instrument 
provides a means of identifying a teacher’s beliefs and intentions regarding the use of technology in teaching and 
learning. The Classroom Observation instrument provides an independent observation concerning how these 
intentions and beliefs are made manifest in practice within the classroom. The Teacher Classroom Observation 
Report confirms the classroom observation. The Survey of Technology Infrastructure instrument provides a look at 
the infrastructure that supports technology use. The Teacher Technology Use instrument describes how deeply 
technology has penetrated into the work of the teacher. The Student Technology Use instrument indicates how 
much technology has entered into the work and play of students and how confident they are in its use. 
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Researchers in schools, government organizations, and private foundations are all searching for a way to 
document the impact of technology on student learning outcomes. These instruments, by measuring the variability' 
of classroom learning environments with respect to technology integration and engaged learning characteristics, and 
by providing individual student measures of technology use, will allow researchers to build and test models of 
student learning with technology as a mediating factor. 

Instrument Use 

• Classroom Observation Instrument. 

This instrument can be used to describe specific classroom learning environments, as a check of reported classroom 
practices, and as a pre-post- assessment of technology usage. There are 21 indicators in total which focus on 
observable classroom artifacts and practices. Use of this instrument requires a trained observer and some means of 
recording scores. If used as part of a district-wide project, a bar code scanner is recommended for its ease of use and 
the elimination of the necessity to enter data into digital format 

• Teacher Beliefs and Practices. 

This instrument is designed to assess the teachers’ pedagogical beliefs and typical classroom practices. In particular, 
it addresses issues that are not possible to observe in one classroom visit, e.g. What are the different ways groups are 
formed for collaborative work? This instrument could be used as part of a pre-post assessment of a training program 
designed to change these teacher practices. 

• Teacher Technology Use. 

The technology use survey is designed to measure the depth of penetration of technology into the teacher’s work. It 
looks at the volume of time allocated to technology use as well as the proportion of units taught which include a 
variety of uses of technology. This instrument could be used as a pre-post assessment of teacher use of technology 
in a program of professional development and/or mentoring designed to increase technology integration. 

• Survey of Technology Infrastructure 

This instrument is used as part of the Engaged Learning/Technology assessment battery. It needs to be completed 
for each school involved in the study. It may be completed by the district tech coordinator or building level 
information specialist/tech coordinator. This instrument is used to collect data regarding the technology 
infrastructure that supports classroom integration of technology. The. ratings from this instrument will be the same 
for all classrooms within the school unit that have access to its resources. 
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• Teacher Report of Classroom Observation 

The teacher report is a validation of the Classroom Observation and can be used as the basis for a conference if great 
disparity is noticed between the two reports of a single classroom activity. 

• Student Technology Use Survey 

This instrument measures the penetration of technology into the life of the student by asking how many times a 
week and how many hours a week the student uses computers or graphing calculators for specific types of activity. 

The instrument also includes a sub-scale for self-efficacy with computers and graphing calculators. The results of 
this instrument can be used to show the variation in student experience and confidence in using computers and 
graphing calculators. 

To obtain a matrix score of engaged/passive learning and high/low technology (Jones, Valdez, 
Nowakowski, & Rasmussen, (1994), four instruments must be used: Classroom Observation Instrument, Teacher 
Beliefs and Practices, Teacher Technology Use Survey, and the Survey of Technology Infrastructure. 

Specimen Set 

The following are sample questions taken from each of the Engaged Leaming/Technology instruments. They are intended to 
show the type of questions asked and the scope of each instrument. 

Classroom Observation Instrument 

C01 - ACTIVE ENGAGEMENT 

Are the teacher/student, student/student actively engaged with each other and with instructional resources? 

Is the teacher creating learning opportunities that stimulate thought and enquiry? [E.g. (including, but not 
limited to the following) linking to external sources of info such as museums, other students, experts, 
examining contrasting data or viewpoints, use of rich media sources to create presentations - images, 
audio, video, 3-D, virtual reality,] 

C09 TEACHER/FACILITATOR 

Is technology being used by the teacher to create a learning environment where he/she is a facilitator? The 
facilitator role will be evidenced by the teacher giving the assignment, providing or preparing needed 
resources and being available for questions. When a problem occurs, the facilitator asks questions that lead 
a student to the solution of the problem. This would not include teachers who step in and solve the problem 
for the youngster. 
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C 1 1 DEVELOP PRODUCTS 

Are students using technology to develop products they can use or share with others? This could be 
commercials to advertise particular books, databases to share with other students, or posters created on computer to 
illustrate advertising techniques. 

C 1 9 DIFFERENT ACCESS POINTS 

Does software offer different routes through the program, different levels to match learner proficiency? 
Are there different types of help for both novice and experienced teachers and students? 

Teacher Report of Classroom Observation 

C01 - ACTIVE ENGAGEMENT 

Are the teacher/student, student/student actively engaged with each other and with instructional resources? 
Is the teacher creating learning opportunities that stimulate thought and enquiry? [E.g. (including, but not 
limited to the following) linking to external sources of info such as museums, other students, experts, 
examining contrasting data or viewpoints, use of rich media sources to create presentations - images, 
audio, video, 3-D, virtual reality.] 

C09 TEACHER/FACILITATOR 

Is technology being used by the teacher to create a learning environment where he/she is a facilitator? The 
facilitator role will be evidenced by the teacher giving the assignment, providing or preparing needed 
resources and being available for questions. When a problem occurs, the facilitator asks questions that lead 
a student to the solution of the problem. This would not include teachers who step in and solve the problem 
for the youngster. 

C 1 7 DEVELOP PRODUCTS 

Are students using technology to develop products they can use or share with others? This could be 
commercials to advertise particular books, databases to share with other students, or posters created on computer to 
illustrate advertising techniques. 

C 1 9 DIFFERENT ACCESS POINTS 

Does software offer different routes through the program, different levels to match learner proficiency? 
Are there different types of help for both novice and experienced teachers and students? 
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Survey of Technology Infrastrucure 

Access 

1 . What type of network connections are present in the school? Please check all that apply. 

No network connections present 

LocalTalk network, 235 Kbps. 

EtherTalk (10 Mbs ethemet), lOBase T (data plus phone) 

Ethemet, lOObaseT 

Other, please explain 



Operability 

T What is the capability to exchange data among diverse technologies used? Please select the BEST answer. 

No provision for data to be exchanged among diverse technologies. 

Students & teachers use “text only” format to enable data exchange. 

All computers run same cross-platform programs (Word, Excel, etc.) 

In addition to using cross platform applications, translation software is available on 

all computers, (e.g. MacLink, Conversions Plus) 

Resource organization? 

1. Are resources centralized or decentralized (distributed)? (resources such as CD-ROMs, videos, hardware, 
software, others?) Please check BEST answer. 

Stand alone computers only, software loaded on as-needed basis, Tech Coordinator 

controls access to software 

Centralized resources, have LAN software collected in media center; may or may not 

have Internet connection in media center and/or labs. 

LAN and WWW, teachers may have folders on server; teachers may have content- 

specific software; classrooms, media centers, and labs are connected to WWW 

LAN, WAN, and WWW, teachers and students have folders on server; CD Towers 

enable simultaneous sharing of resources by a number of classrooms; classrooms and 
schools connected through LAN and WAN. Internet available at every computer 

other, please explain 



Teacher Beliefs and Practices 

1. How are your students involved in setting goals, developing assessments and setting standards for learning tasks? 
Please check the BEST answer. 

Goals primarily set through curriculum/teacher 

Mixture of student, teacher, and curriculum used to set goals, students may 

participate in developing rubrics used for assessment. 

Teacher facilitates as students decide how to meet curriculum guide lines 

students devise activities and rubrics to asses them 
Other, please explain 



5. How are learning tasks selected? Please check the BEST answer. 

Teacher selects all tasks or uses textbook; same tasks for everyone. 
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Use curriculum guidelines strictly, may consider student preferences, 

strengths, weaknesses, etc. May include choice between alternatives. 

Students have input in selecting tasks for learning; may choose from a menu 

of alternatives, or devise own activities. Curriculum used as a guide. 

__ Other, please explain 

8. Describe tasks/projects that required information from another discipline for completion. Please check the BEST 
answer. 



No planning for cross discipline studies 

Specific tasks may use skills from other disciplines(e.g. writing in 
mathematics or using measuring skills in chemistry 
Small thematic projects are planned that incorporate other disciplines 
(e.g. reports requiring library and writing skills, book reports in history class, 
creating databases in social studies.) 

Major learning segments of the class are thematic units or problem-based 
learning that require the integration of multiple areas of study, (e.g. 

Designing a zoo including the requirements for food, shelter, companionship, 
exercise, etc. for each animal.) 

Other, please explain 
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Survey of Technology Infrastructure 



Access 

1 . What type of network connections are present in the school? Please check all that apply. 

No network connections present 

LocalTalk network, 235 Kbps. 

EtherTalk (10 Mbs ethemet), lOBase T (data plus phone) 

Ethemet, lOObase T 

Other, please explain 

2. What type of Internet connection is used by the school? Please check BEST answer. 

No Internet 

Dial-up connection 

56 K dedicated line 

T1 or higher dedicated line 

Other, please explain 

3. How many connections to the Internet can be sustained simultaneously? Please check 
BEST answer. 

No connections 

Some (at least 1) computer(s) in the school have access 

Nearly every computer in school has access 

At least every computer in school is connected, may have excess capacity 

4. How far do teachers and students have to go to access diverse technologies (e.g. 
computer, printer, scanner, Internet connection, digital camera)? (classroom, next floor, 
next building, media center?) Please check BEST answer. 

None available 

Available within the district 

Available within the building 

All technologies available in the classroom or nearby 

5. Briefly describe the extent to which technology is being used at this school. Please 
check BEST answer. 

No use of technology 

Few proficient users, more non-users 

Many emergent users, some proficient, some reluctant 

Many proficient users, no non-users 

6. What provisions have been made to ensure equitable use/access of hardware and 
software for all students and teachers? Please check BEST answer. 

Not addressed 

Teachers responsible for ensuring students have equal access to technologies. 

Policy/plan for ensuring equal access is in place. 

Well-established policies are applied to ensure equitable access to students including 

open labs before, after, and during school times. May have laptops or word 
processors for check-out to students. 
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7, Do students who are not enrolled in computer based courses have the same access as 
those who are? Please check the BEST answer. 

No access outside of class time. 

Limited access (short periods of time before/after school ; may be media center only) 

Well established access (labs/classrooms/& media center open at least 1 hr before/after 

school, most school computers included.) 

Other, please explain 

Operability 

1 . What is the capability to exchange data among diverse technologies used? Please select 
the BEST answer. 

No provision for data to be exchanged among diverse technologies. 

Students & teachers use “text only” format to enable data exchange. 

All computers run same cross-platform programs (Word, Excel, etc.) 

In addition to using cross platform applications, translation software is available on 

all computers, (e.g. MacLink, Conversions Plus) 

2. Can groups of teachers or students share documents and data simultaneously? Please 
check BEST answer. 

No 

Yes, limited availability 

Yes, not often used 

Yes, commonly, used 

Other, please explain 

3. Describe the periods of downtime delays experienced by the system. Please check 
BEST answer. 

Down more than working, difficult to use 

Frequently down, sluggish, long repair times 

May occasionally be down, or sometime take awhile to fix 

Rarely down, readily repaired 

4. Are users able to operate the system easily? Please select the BEST answer. 

No 

Yes, some 

Yes. many 

Yes, most 

Resource organization? 

1. Are resources centralized or decentralized (distributed)? (resources such as CD-ROMs, 
videos, hardware, software, others?) Please check BEST answer. 

Stand alone computers only, software loaded on as-needed basis, Tech Coordinator 

controls access to software 

Centralized resources, have LAN software collected in media center; may or may not 

have Internet connection in media center and/or labs. 

LAN and WWW, teachers may have folders on server; teachers may have content- 

specific software; classrooms, media centers, and labs are connected to WWW 

LAN, WAN, and WWW, teachers and students have folders on server; CD Towers 

enable simultaneous sharing of resources by a number of classrooms; classrooms and 

Slur Schools Evaluation, E006 Lagorrutrcino Hall. Iowa State Universiry, 515-294-6919 
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schools connected through LAN and WAN. Internet available at every computer 
other, please explain 

2. Can users provide resources to the system on demand? Please check BEST answer. 

No resources may be added to the system 

Yes, Tech coordinators only may add information 

Yes, staff may add information 

Yes, staff and students may freely add information 

Other, please explain 

3. Is the system designed to facilitate communication among users with diverse 
technologies to carry out collaborative projects? Please check BEST answer. 

No 

System is capable of use for collaborative projects, but not enabled 

System is used for collaborative projects by a select group or used sometimes 

Yes, system freely available for collaborative projects 

Other, please explain 
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Clearinghouse on Assessment and Evaluation 



University of Maryland 
1129 Shriver Laboratory 
College Park, MD 20742-5701 

Tel: (800) 464-3742 
(301)405-7449 
FAX: (301) 405-8134 
ericae @ ericae. net 
http://ericae.net 

Congratulations on being a presenter at AERA. The ERIC Clearinghouse on Assessment and 
Evaluation would like you to contribute to ERIC by providing us with a written copy of your 
presentation. Submitting your paper to ERIC ensures a wider audience by making it available to 
members of the education community who could not attend your session or this year's conference. 

Abstracts of papers accepted by ERIC appear in Resources in Education ( RIE ') and are announced to over 
5,000 organizations. The inclusion of your work makes it readily available to other researchers, provides a 
permanent archive, and enhances the quality of RIE. Abstracts of your contribution will be accessible 
through the printed, electronic, and internet versions of RIE. The paper will be available full-text, on 
demand through the ERIC Document Reproduction Service and through the microfiche collections 
housed at libraries around the world. 

We are gathering all the papers from the AERA Conference. We will route your paper to the 
appropriate clearinghouse and you will be notified if your paper meets ERIC's criteria. Documents 
are reviewed for contribution to education, timeliness, relevance, methodology, effectiveness of 
presentation, and reproduction quality. You can track our processing of your paper at 

http://ericae.net. 

To disseminate your work through ERIC, you need to sign the reproduction release form on the 
back of this letter and include it with two copies of your paper. You can drop of the copies of 
your paper and reproduction release form at the ERIC booth (223) or mail to our attention at the 
address below. If you have not submitted your 1999 Conference paper please send today or 
drop it off at the booth with a Reproduction Release Form. Please feel free to copy the form 
for future or additional submissions. 

Mail to: AERA 2000/ERIC Acquisitions 

The University of Maryland 
1129 Shriver Lab 
College Park, MD 20742 



Sincerely, 




Lawrence M. Rudner, Ph.D. 
Director, ERIC/AE 



March 2000 

Dear AERA Presenter, 





ERIC/AE is a project of the Department of Measurement, Statistics and Evaluation 
at the College of Education, University of Maryland. 




